Allow proc_mesh.stop() after actor_mesh.stop() #877
Closed
+23
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D80269385 |
thomasywang
force-pushed
the
export-D80269385
branch
from
c306dcd to
4077005
Compare
thomasywang
added a commit
to thomasywang/monarch-1
that referenced
this pull request
Aug 15, 2025
Summary: Currently calling proc_mesh.stop() will fail if an actor_mesh spawned on this proc_mesh has been stopped. This is because the python-binding wrapper for ProcMesh (TrackedProcMesh) holds onto a map of SharedCells containing each RootActorMesh spawned on it. When an PythonActorMesh is stopped, the RootActorMesh is dropped, but does not update the state of the TrackedProcMesh to remove this. When a ProcMesh is stopped, it will attempt to discard every SharedCell containing it's RootActorMeshes, but will return an error if any RootActorMesh has been dropped (The error we are seeing) We can make it such that stopping the PythonActorMesh will update the state of the TrackedProcMesh, but this would involve plumbing through references of TrackedProcMesh and related state to PythonActorMesh A simpler solution, it to simply allow the TrackedProcMesh to discard the errors related to attempting to discard a SharedCell where the RootActorMesh has already been dropped Reviewed By: ahmadsharif1 Differential Revision: D80269385
|
This pull request was exported from Phabricator. Differential Revision: D80269385 |
thomasywang
added a commit
to thomasywang/monarch-1
that referenced
this pull request
Aug 15, 2025
Summary: Pull Request resolved: meta-pytorch#877 Currently calling proc_mesh.stop() will fail if an actor_mesh spawned on this proc_mesh has been stopped. This is because the python-binding wrapper for ProcMesh (TrackedProcMesh) holds onto a map of SharedCells containing each RootActorMesh spawned on it. When an PythonActorMesh is stopped, the RootActorMesh is dropped, but does not update the state of the TrackedProcMesh to remove this. When a ProcMesh is stopped, it will attempt to discard every SharedCell containing it's RootActorMeshes, but will return an error if any RootActorMesh has been dropped (The error we are seeing) We can make it such that stopping the PythonActorMesh will update the state of the TrackedProcMesh, but this would involve plumbing through references of TrackedProcMesh and related state to PythonActorMesh A simpler solution, it to simply allow the TrackedProcMesh to discard the errors related to attempting to discard a SharedCell where the RootActorMesh has already been dropped Reviewed By: ahmadsharif1 Differential Revision: D80269385
thomasywang
force-pushed
the
export-D80269385
branch
from
4077005 to
5e3de06
Compare
thomasywang
added a commit
to thomasywang/monarch-1
that referenced
this pull request
Aug 16, 2025
Summary: Currently calling proc_mesh.stop() will fail if an actor_mesh spawned on this proc_mesh has been stopped. This is because the python-binding wrapper for ProcMesh (TrackedProcMesh) holds onto a map of SharedCells containing each RootActorMesh spawned on it. When an PythonActorMesh is stopped, the RootActorMesh is dropped, but does not update the state of the TrackedProcMesh to remove this. When a ProcMesh is stopped, it will attempt to discard every SharedCell containing it's RootActorMeshes, but will return an error if any RootActorMesh has been dropped (The error we are seeing) We can make it such that stopping the PythonActorMesh will update the state of the TrackedProcMesh, but this would involve plumbing through references of TrackedProcMesh and related state to PythonActorMesh A simpler solution, it to simply allow the TrackedProcMesh to discard the errors related to attempting to discard a SharedCell where the RootActorMesh has already been dropped Reviewed By: ahmadsharif1 Differential Revision: D80269385
thomasywang
force-pushed
the
export-D80269385
branch
from
5e3de06 to
caef19a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D80269385 |
thomasywang
added a commit
to thomasywang/monarch-1
that referenced
this pull request
Aug 16, 2025
Summary: Pull Request resolved: meta-pytorch#877 Currently calling proc_mesh.stop() will fail if an actor_mesh spawned on this proc_mesh has been stopped. This is because the python-binding wrapper for ProcMesh (TrackedProcMesh) holds onto a map of SharedCells containing each RootActorMesh spawned on it. When an PythonActorMesh is stopped, the RootActorMesh is dropped, but does not update the state of the TrackedProcMesh to remove this. When a ProcMesh is stopped, it will attempt to discard every SharedCell containing it's RootActorMeshes, but will return an error if any RootActorMesh has been dropped (The error we are seeing) We can make it such that stopping the PythonActorMesh will update the state of the TrackedProcMesh, but this would involve plumbing through references of TrackedProcMesh and related state to PythonActorMesh A simpler solution, it to simply allow the TrackedProcMesh to discard the errors related to attempting to discard a SharedCell where the RootActorMesh has already been dropped Reviewed By: ahmadsharif1 Differential Revision: D80269385
thomasywang
force-pushed
the
export-D80269385
branch
from
caef19a to
8239b40
Compare
thomasywang
added a commit
to thomasywang/monarch-1
that referenced
this pull request
Aug 18, 2025
Summary: Currently calling proc_mesh.stop() will fail if an actor_mesh spawned on this proc_mesh has been stopped. This is because the python-binding wrapper for ProcMesh (TrackedProcMesh) holds onto a map of SharedCells containing each RootActorMesh spawned on it. When an PythonActorMesh is stopped, the RootActorMesh is dropped, but does not update the state of the TrackedProcMesh to remove this. When a ProcMesh is stopped, it will attempt to discard every SharedCell containing it's RootActorMeshes, but will return an error if any RootActorMesh has been dropped (The error we are seeing) We can make it such that stopping the PythonActorMesh will update the state of the TrackedProcMesh, but this would involve plumbing through references of TrackedProcMesh and related state to PythonActorMesh A simpler solution, it to simply allow the TrackedProcMesh to discard the errors related to attempting to discard a SharedCell where the RootActorMesh has already been dropped Reviewed By: ahmadsharif1 Differential Revision: D80269385
thomasywang
force-pushed
the
export-D80269385
branch
from
8239b40 to
cf09cea
Compare
Summary: Pull Request resolved: meta-pytorch#877 Currently calling proc_mesh.stop() will fail if an actor_mesh spawned on this proc_mesh has been stopped. This is because the python-binding wrapper for ProcMesh (TrackedProcMesh) holds onto a map of SharedCells containing each RootActorMesh spawned on it. When an PythonActorMesh is stopped, the RootActorMesh is dropped, but does not update the state of the TrackedProcMesh to remove this. When a ProcMesh is stopped, it will attempt to discard every SharedCell containing it's RootActorMeshes, but will return an error if any RootActorMesh has been dropped (The error we are seeing) We can make it such that stopping the PythonActorMesh will update the state of the TrackedProcMesh, but this would involve plumbing through references of TrackedProcMesh and related state to PythonActorMesh A simpler solution, it to simply allow the TrackedProcMesh to discard the errors related to attempting to discard a SharedCell where the RootActorMesh has already been dropped Reviewed By: ahmadsharif1 Differential Revision: D80269385
|
This pull request was exported from Phabricator. Differential Revision: D80269385 |
thomasywang
force-pushed
the
export-D80269385
branch
from
cf09cea to
57b15d7
Compare
|
This pull request has been merged in cac2e47. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Currently calling proc_mesh.stop() will fail if an actor_mesh spawned on this proc_mesh has been stopped.
This is because the python-binding wrapper for ProcMesh (TrackedProcMesh) holds onto a map of SharedCells containing each RootActorMesh spawned on it. When an PythonActorMesh is stopped, the RootActorMesh is dropped, but does not update the state of the TrackedProcMesh to remove this. When a ProcMesh is stopped, it will attempt to discard every SharedCell containing it's RootActorMeshes, but will return an error if any RootActorMesh has been dropped (The error we are seeing)
We can make it such that stopping the PythonActorMesh will update the state of the TrackedProcMesh, but this would involve plumbing through references of TrackedProcMesh and related state to PythonActorMesh
A simpler solution, it to simply allow the TrackedProcMesh to discard the errors related to attempting to discard a SharedCell where the RootActorMesh has already been dropped
Differential Revision: D80269385